{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "# Using Feature Store\n",
        "\n",
        "## Learning Objective\n",
        "\n",
        "In this notebook, you will learn how to:\n",
        "\n",
        "* Import your features into Feature Store.\n",
        "* Serve online prediction requests using the imported features.\n",
        "* Access imported features in offline jobs, such as training jobs.\n",
        "\n",
        "## Introduction\n",
        "\n",
        "In this notebook, you will learn how to use Feature Store, a managed cloud service for machine learning engineers and data scientists to store, serve, manage and share machine learning features at a large scale.\n",
        "\n",
        "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/6_gapic_feature_store.ipynb) -- try to complete that notebook first before reviewing this solution notebook. \n",
        "\n",
        "This notebook assumes that you understand basic Google Cloud concepts such as [Project](https://cloud.google.com/storage/docs/projects), [Storage](https://cloud.google.com/storage) and [Vertex AI](https://cloud.google.com/vertex-ai/docs). Some machine learning knowledge is also helpful but not required.\n",
        "\n",
        "### Dataset\n",
        "\n",
        "This notebook uses a movie recommendation dataset as an example throughout all the sessions. The task is to train a model to predict if a user is going to watch a movie and serve this model online. \n",
        "\n",
        "**Make sure to enable the Vertex AI, Cloud Storage, and Compute Engine APIs.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install additional packages\n",
        "\n",
        "For this notebook, you need the Vertex SDK for Python."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "2b4ef9b72d43",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Setup your dependencies\n",
        "import os\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
        "USER_FLAG = \"\"\n",
        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    USER_FLAG = \"--user\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wyy5Lbnzg5fi",
        "outputId": "a804bccf-b2f4-41c2-f3f0-a206b2bb0899",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting git+https://github.com/googleapis/python-aiplatform.git@main-test\n",
            "  Cloning https://github.com/googleapis/python-aiplatform.git (to revision main-test) to /tmp/pip-req-build-wm84ezm8\n",
            "  Running command git clone --filter=blob:none -q https://github.com/googleapis/python-aiplatform.git /tmp/pip-req-build-wm84ezm8\n",
            "  Running command git checkout -b main-test --track origin/main-test\n",
            "  Switched to a new branch 'main-test'\n",
            "  Branch 'main-test' set up to track remote branch 'main-test' from 'origin'.\n",
            "  Resolved https://github.com/googleapis/python-aiplatform.git to commit 59808e63aa07a725de03721ecdd4efef9e1b33e2\n",
            "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25hCollecting google-api-core[grpc]<2.0.0dev,>=1.22.2\n",
            "  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)\n",
            "     |████████████████████████████████| 93 kB 1.7 MB/s             \n",
            "\u001b[?25hRequirement already satisfied: proto-plus>=1.10.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==0.7.1) (1.19.8)\n",
            "Requirement already satisfied: google-cloud-storage<2.0.0dev,>=1.26.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==0.7.1) (1.43.0)\n",
            "Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==0.7.1) (2.31.0)\n",
            "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (21.3)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (1.53.0)\n",
            "Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (2021.3)\n",
            "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (3.19.1)\n",
            "Requirement already satisfied: six>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (1.16.0)\n",
            "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (59.6.0)\n",
            "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (2.26.0)\n",
            "Collecting google-auth<2.0dev,>=1.25.0\n",
            "  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
            "     |████████████████████████████████| 152 kB 22.9 MB/s            \n",
            "\u001b[?25hRequirement already satisfied: grpcio<2.0dev,>=1.29.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (1.43.0)\n",
            "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==0.7.1) (2.8.2)\n",
            "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==0.7.1) (2.1.0)\n",
            "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==0.7.1) (2.2.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (0.2.7)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (4.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (4.2.4)\n",
            "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==0.7.1) (1.1.2)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (3.0.6)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (1.26.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (2.0.9)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (3.1)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (2021.10.8)\n",
            "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==0.7.1) (1.15.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-aiplatform==0.7.1) (0.4.8)\n",
            "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==0.7.1) (2.21)\n",
            "Building wheels for collected packages: google-cloud-aiplatform\n",
            "  Building wheel for google-cloud-aiplatform (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Created wheel for google-cloud-aiplatform: filename=google_cloud_aiplatform-0.7.1-py2.py3-none-any.whl size=1696810 sha256=c7f7592b798b8956fd0909a0f6685e0a317b980909919c4c108fc9dbfad521ca\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-klf3ea77/wheels/17/ff/e0/00d0fec24bcb16097ce47b6894c6844b347b9fc2b413623116\n",
            "Successfully built google-cloud-aiplatform\n",
            "Installing collected packages: google-auth, google-api-core, google-cloud-aiplatform\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "explainable-ai-sdk 1.3.2 requires xai-image-widget, which is not installed.\n",
            "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 2.33.0 which is incompatible.\u001b[0m\n",
            "Successfully installed google-api-core-1.31.5 google-auth-1.35.0 google-cloud-aiplatform-0.7.1\n"
          ]
        }
      ],
      "source": [
        "# Upgrade the specified package to the newest available version\n",
        "! pip3 install {USER_FLAG} --upgrade git+https://github.com/googleapis/python-aiplatform.git@main-test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Note**: Please ignore any incompatibility warnings and errors.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hhq5zEbGg0XX"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "After you install the SDK, you need to restart the notebook kernel so it can find the packages. You can restart kernel from *Kernel -> Restart Kernel*, or running the following:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "EzrelQZ22IZj",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs\n",
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "Update `YOUR-PROJECT-ID` with your Project ID. **If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oM1iC_MfAts1",
        "outputId": "83b174cb-5cdc-45f1-b67a-152f84a9fc24",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Project ID:  qwiklabs-gcp-01-3ea79ecd1cfe\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "PROJECT_ID = \"YOUR-PROJECT-ID\"\n",
        "\n",
        "# Get your Google Cloud project ID from gcloud\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    # PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID: \", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJYoRfYng0XZ"
      },
      "source": [
        "Otherwise, set your project ID here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "riG_qUokg0XZ",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
        "    PROJECT_ID = \"YOUR-PROJECT-ID\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. It must be unique across all of your \n",
        "Cloud Storage buckets.\n",
        "\n",
        "You may also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
        "available](https://cloud.google.com/vertex-ai/docs/general/locations). You may\n",
        "not use a Multi-Regional Storage bucket for training with Vertex AI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://<YOUR-BUCKET-NAME>\"  # @param {type:\"string\"}\n",
        "REGION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "! gcloud storage buckets create --location=$REGION $BUCKET_NAME"      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "! gcloud storage buckets update --uniform-bucket-level-access gs://<YOUR-BUCKET-NAME>"      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "PyQmSRbKA8r-",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Authenticate your google Cloud account\n",
        "import os\n",
        "import sys\n",
        "\n",
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# If on Google Cloud Notebooks, then don't execute this code\n",
        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    if \"google.colab\" in sys.modules:\n",
        "        from google.colab import auth as google_auth\n",
        "\n",
        "        google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    elif not os.getenv(\"IS_TESTING\"):\n",
        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gAL9Y4VTOLT1"
      },
      "source": [
        "## Prepare for output\n",
        "\n",
        "### Step 1. Create dataset for output\n",
        "\n",
        "You need a BigQuery dataset to host the output data in `us-central1`. Input the name of the dataset you want to create and specify the name of the table you want to store the output later. These will be used later in the notebook.\n",
        "\n",
        "**Make sure that the table name does NOT already exist**.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "4Oc-jrd6Ow7N",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "from google.cloud import bigquery"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "QOaGi2PrEAwA",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Output dataset\n",
        "DESTINATION_DATA_SET = \"movie_predictions\"  # @param {type:\"string\"}\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
        "DESTINATION_DATA_SET = \"{prefix}_{timestamp}\".format(\n",
        "    prefix=DESTINATION_DATA_SET, timestamp=TIMESTAMP\n",
        ")\n",
        "\n",
        "# Output table. Make sure that the table does NOT already exist; the BatchReadFeatureValues API cannot overwrite an existing table\n",
        "DESTINATION_TABLE_NAME = \"training_data\"  # @param {type:\"string\"}\n",
        "\n",
        "DESTINATION_PATTERN = \"bq://{project}.{dataset}.{table}\"\n",
        "DESTINATION_TABLE_URI = DESTINATION_PATTERN.format(\n",
        "    project=PROJECT_ID, dataset=DESTINATION_DATA_SET, table=DESTINATION_TABLE_NAME\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RKhmymT-O0vy",
        "outputId": "80e93949-3d46-461c-b67f-18d2460bd7ca",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Created dataset qwiklabs-gcp-01-3ea79ecd1cfe.movie_predictions_20210824110930\n"
          ]
        }
      ],
      "source": [
        "# Create dataset\n",
        "REGION = \"us-central1\"  # @param {type:\"string\"}\n",
        "client = bigquery.Client()\n",
        "dataset_id = \"{}.{}\".format(client.project, DESTINATION_DATA_SET)\n",
        "dataset = bigquery.Dataset(dataset_id)\n",
        "dataset.location = REGION\n",
        "dataset = client.create_dataset(dataset, timeout=30)\n",
        "print(\"Created dataset {}.{}\".format(client.project, dataset.dataset_id))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XoEqT2Y4DJmf"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Copying gs://cloud-training/mlongcp/v3.0_MLonGC/toy_data/feature_stores/movie_prediction_toy.csv [Content-Type=text/csv]...\n",
            "Copying gs://cloud-training/mlongcp/v3.0_MLonGC/toy_data/feature_stores/movie_prediction_toy.csv [Content-Type=text/csv]...\n",
            "Copying gs://cloud-training/mlongcp/v3.0_MLonGC/toy_data/feature_stores/movie_prediction_toy.csv [Content-Type=text/csv]...\n",
            "/ [3 files][  1.5 KiB/  1.5 KiB]                                                \n",
            "Operation completed over 3 objects/1.5 KiB.                                      \n"
          ]
        }
      ],
      "source": [
        "# Copy all required files in your bucket.\n",
        "# Make sure to replace your bucket name here.\n",
        "!gcloud storage cp --recursive gs://cloud-training/mlongcp/v3.0_MLonGC/toy_data/feature_stores/* gs://<YOUR-BUCKET-NAME>"      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "isNzmylQXjly",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Other than project ID and featurestore ID and endpoints needs to be set.\n",
        "# Make sure to replace your bucket name here.\n",
        "API_ENDPOINT = \"us-central1-aiplatform.googleapis.com\"  # @param {type:\"string\"}\n",
        "INPUT_CSV_FILE = \"gs://<YOUR-BUCKET-NAME>/movie_prediction_toy.csv\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "pRUOFELefqf1",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from google.cloud.aiplatform_v1beta1 import (\n",
        "    FeaturestoreOnlineServingServiceClient, FeaturestoreServiceClient)\n",
        "from google.cloud.aiplatform_v1beta1.types import FeatureSelector, IdMatcher\n",
        "from google.cloud.aiplatform_v1beta1.types import \\\n",
        "    entity_type as entity_type_pb2\n",
        "from google.cloud.aiplatform_v1beta1.types import feature as feature_pb2\n",
        "from google.cloud.aiplatform_v1beta1.types import \\\n",
        "    featurestore as featurestore_pb2\n",
        "from google.cloud.aiplatform_v1beta1.types import \\\n",
        "    featurestore_monitoring as featurestore_monitoring_pb2\n",
        "from google.cloud.aiplatform_v1beta1.types import \\\n",
        "    featurestore_online_service as featurestore_online_service_pb2\n",
        "from google.cloud.aiplatform_v1beta1.types import \\\n",
        "    featurestore_service as featurestore_service_pb2\n",
        "from google.cloud.aiplatform_v1beta1.types import io as io_pb2\n",
        "from google.protobuf.duration_pb2 import Duration\n",
        "\n",
        "# Create admin_client for CRUD and data_client for reading feature values.\n",
        "admin_client = FeaturestoreServiceClient(client_options={\"api_endpoint\": API_ENDPOINT})\n",
        "data_client = FeaturestoreOnlineServingServiceClient(\n",
        "    client_options={\"api_endpoint\": API_ENDPOINT}\n",
        ")\n",
        "\n",
        "# Represents featurestore resource path.\n",
        "BASE_RESOURCE_PATH = admin_client.common_location_path(PROJECT_ID, REGION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_HmF24mBHv9"
      },
      "source": [
        "## Terminology and Concept\n",
        "\n",
        "### Featurestore Data model\n",
        "\n",
        "Feature Store organizes data with the following 3 important hierarchical concepts:\n",
        "```\n",
        "Featurestore -> EntityType -> Feature\n",
        "```\n",
        "* **Featurestore**: the place to store your features\n",
        "* **EntityType**: under a Featurestore, an *EntityType* describes an object to be modeled, real one or virtual one.\n",
        "* **Feature**: under an EntityType, a *feature* describes an attribute of the EntityType\n",
        "\n",
        "In the movie prediction example, you will create a featurestore called *movie_prediction*. This store has 2 entity types: *Users* and *Movies*. The Users entity type has the age, gender, and like genres features. The Movies entity type has the genres and average rating features.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UvxYyGUimKw"
      },
      "source": [
        "## Create Featurestore and Define Schemas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "buQBIv3ZL3A0"
      },
      "source": [
        "### Create Featurestore\n",
        "\n",
        "The method to create a featurestore returns a\n",
        "[long-running operation](https://google.aip.dev/151) (LRO). An LRO starts an asynchronous job. LROs are returned for other API\n",
        "methods too, such as updating or deleting a featurestore. Calling\n",
        "`create_fs_lro.result()` waits for the LRO to complete."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "FscHZa0DXjmC",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "FEATURESTORE_ID = \"movie_prediction_{timestamp}\".format(timestamp=TIMESTAMP)\n",
        "create_lro = admin_client.create_featurestore(\n",
        "    featurestore_service_pb2.CreateFeaturestoreRequest(\n",
        "        parent=BASE_RESOURCE_PATH,\n",
        "        featurestore_id=FEATURESTORE_ID,\n",
        "        featurestore=featurestore_pb2.Featurestore(\n",
        "            display_name=\"Featurestore for movie prediction\",\n",
        "            online_serving_config=featurestore_pb2.Featurestore.OnlineServingConfig(\n",
        "                fixed_node_count=3\n",
        "            ),\n",
        "        ),\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "57V8eVcB5VFZ",
        "outputId": "ac88a457-fa17-4702-a18b-587b0def028d",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930\"\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Wait for LRO to finish and get the LRO result.\n",
        "print(create_lro.result())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ag8pCQ7rNjVf"
      },
      "source": [
        "You can use [GetFeaturestore](https://cloud.google.com/vertex-ai/docs/featurestore) or [Featurestores](https://cloud.google.com/vertex-ai/docs/featurestore/managing-featurestores) to check if the Featurestore was successfully created. The following example gets the details of the Featurestore.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eKhD4q8rXjmM",
        "outputId": "0b4aabe3-1e81-4dca-a547-c340fbfa3706",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930\"\n",
              "create_time {\n",
              "  seconds: 1629803788\n",
              "  nanos: 111037000\n",
              "}\n",
              "update_time {\n",
              "  seconds: 1629803788\n",
              "  nanos: 187222000\n",
              "}\n",
              "etag: \"AMEw9yPmZHh0tF82VsW42YlQCrGd-AbPbMYwR6_WS0tHdthJCHIIixMkivjGGFGko14=\"\n",
              "online_serving_config {\n",
              "  fixed_node_count: 3\n",
              "}\n",
              "state: STABLE"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "admin_client.get_featurestore(\n",
        "    name=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EpmJq75zXjmT"
      },
      "source": [
        "### Create Entity Type\n",
        "You can specify a monitoring config which will by default be inherited by all Features under this EntityType."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s9eZ7aJLXjmT",
        "outputId": "34b868f2-5154-4205-f33f-15ef6f7fc290",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users\"\n",
            "etag: \"AMEw9yNw3iflwyrbZyyI03Vx09aaI-KAS214bwrKzEGvEF3RSClb\"\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Create users entity type with monitoring enabled.\n",
        "# All Features belonging to this EntityType will by default inherit the monitoring config.\n",
        "users_entity_type_lro = admin_client.create_entity_type(\n",
        "    featurestore_service_pb2.CreateEntityTypeRequest(\n",
        "        parent=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
        "        entity_type_id=\"users\",\n",
        "        entity_type=entity_type_pb2.EntityType(\n",
        "            description=\"Users entity\",\n",
        "            monitoring_config=featurestore_monitoring_pb2.FeaturestoreMonitoringConfig(\n",
        "                snapshot_analysis=featurestore_monitoring_pb2.FeaturestoreMonitoringConfig.SnapshotAnalysis(\n",
        "                    monitoring_interval=Duration(seconds=86400),  # 1 day\n",
        "                ),\n",
        "            ),\n",
        "        ),\n",
        "    )\n",
        ")\n",
        "\n",
        "# Similarly, wait for EntityType creation operation.\n",
        "print(users_entity_type_lro.result())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hJqNOttvOc2a",
        "outputId": "f5bd00f2-d8b1-4af0-d211-ba5f199a163c",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies\"\n",
            "etag: \"AMEw9yOxjDQ45socCP2SLRtRmxvAfa0oqDd78VUrmQKuLi4no0pM\"\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Create movies entity type without a monitoring configuration.\n",
        "movies_entity_type_lro = admin_client.create_entity_type(\n",
        "    featurestore_service_pb2.CreateEntityTypeRequest(\n",
        "        parent=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
        "        entity_type_id=\"movies\",\n",
        "        entity_type=entity_type_pb2.EntityType(description=\"Movies entity\"),\n",
        "    )\n",
        ")\n",
        "\n",
        "# Similarly, wait for EntityType creation operation.\n",
        "print(movies_entity_type_lro.result())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FJW4q-0jO2Xf"
      },
      "source": [
        "### Create Feature\n",
        "\n",
        "You can also set a custom monitoring configuration at the Feature level, and view the properties and metrics in the console: sample [properties](https://storage.googleapis.com/cloud-samples-data/ai-platform-unified/datasets/featurestore/Feature%20Properties.png), sample [metrics](https://storage.googleapis.com/cloud-samples-data/ai-platform-unified/datasets/featurestore/Feature%20Snapshot%20Distribution.png)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZJD7-6GFqc1z",
        "outputId": "01ab4688-76f9-4f3c-9ee3-ff9fabf542e6",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "features {\n",
              "  name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users/features/age\"\n",
              "  etag: \"AMEw9yNhKHWFVrNDy8UGUb6bB-lUtt9uiJEvPYU654L3b6g8C1jB\"\n",
              "}\n",
              "features {\n",
              "  name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users/features/gender\"\n",
              "  etag: \"AMEw9yOxR6SMwJ8Viv2_OHWVBofxqe9IYhjItl5kAyb830bQ4rCS\"\n",
              "}\n",
              "features {\n",
              "  name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users/features/liked_genres\"\n",
              "  etag: \"AMEw9yN7epHeP-iZQzzDViV_F7r0rD8Vuta6DTu486dqcwA0o5qD\"\n",
              "}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Create features for the 'users' entity.\n",
        "# 'age' Feature leaves the monitoring config unset, which means it'll inherit the config from EntityType.\n",
        "# 'gender' Feature explicitly disables monitoring.\n",
        "# 'liked_genres' Feature is a STRING_ARRAY type, so it is automatically excluded from monitoring.\n",
        "# For Features with monitoring enabled, distribution statistics are updated periodically in the console.\n",
        "admin_client.batch_create_features(\n",
        "    parent=admin_client.entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"),\n",
        "    requests=[\n",
        "        featurestore_service_pb2.CreateFeatureRequest(\n",
        "            feature=feature_pb2.Feature(\n",
        "                value_type=feature_pb2.Feature.ValueType.INT64,\n",
        "                description=\"User age\",\n",
        "            ),\n",
        "            feature_id=\"age\",\n",
        "        ),\n",
        "        featurestore_service_pb2.CreateFeatureRequest(\n",
        "            feature=feature_pb2.Feature(\n",
        "                value_type=feature_pb2.Feature.ValueType.STRING,\n",
        "                description=\"User gender\",\n",
        "                monitoring_config=featurestore_monitoring_pb2.FeaturestoreMonitoringConfig(\n",
        "                    snapshot_analysis=featurestore_monitoring_pb2.FeaturestoreMonitoringConfig.SnapshotAnalysis(\n",
        "                        disabled=True,\n",
        "                    ),\n",
        "                ),\n",
        "            ),\n",
        "            feature_id=\"gender\",\n",
        "        ),\n",
        "        featurestore_service_pb2.CreateFeatureRequest(\n",
        "            feature=feature_pb2.Feature(\n",
        "                value_type=feature_pb2.Feature.ValueType.STRING_ARRAY,\n",
        "                description=\"An array of genres that this user liked\",\n",
        "            ),\n",
        "            feature_id=\"liked_genres\",\n",
        "        ),\n",
        "    ],\n",
        ").result()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tWx_wI_FS8tE",
        "outputId": "74e61dd0-da7d-4cdc-f31a-bd1c9a612264",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "features {\n",
              "  name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/title\"\n",
              "  etag: \"AMEw9yPsumOeGiCRw4hISl6ABAiD7qSfyEetva9bpmn5qhXM9Pw-\"\n",
              "}\n",
              "features {\n",
              "  name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/genres\"\n",
              "  etag: \"AMEw9yOwr-bHUVgv625Q_xDrXGIoQe9XAP0YakXx9gkjUfXCwJxv\"\n",
              "}\n",
              "features {\n",
              "  name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/average_rating\"\n",
              "  etag: \"AMEw9yMB2lIFR-wjjSXRvpyonp8k4t7LDLIzHSQWvRyt-ruKn-Bv\"\n",
              "}"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Create features for movies type.\n",
        "# 'title' Feature enables monitoring.\n",
        "admin_client.batch_create_features(\n",
        "    parent=admin_client.entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, \"movies\"),\n",
        "    requests=[\n",
        "        featurestore_service_pb2.CreateFeatureRequest(\n",
        "            feature=feature_pb2.Feature(\n",
        "                value_type=feature_pb2.Feature.ValueType.STRING,\n",
        "                description=\"The title of the movie\",\n",
        "                monitoring_config=featurestore_monitoring_pb2.FeaturestoreMonitoringConfig(\n",
        "                    snapshot_analysis=featurestore_monitoring_pb2.FeaturestoreMonitoringConfig.SnapshotAnalysis(\n",
        "                        monitoring_interval=Duration(seconds=172800),  # 2 days\n",
        "                    ),\n",
        "                ),\n",
        "            ),\n",
        "            feature_id=\"title\",\n",
        "        ),\n",
        "        featurestore_service_pb2.CreateFeatureRequest(\n",
        "            feature=feature_pb2.Feature(\n",
        "                value_type=feature_pb2.Feature.ValueType.STRING,\n",
        "                description=\"The genres of the movie\",\n",
        "            ),\n",
        "            feature_id=\"genres\",\n",
        "        ),\n",
        "        featurestore_service_pb2.CreateFeatureRequest(\n",
        "            feature=feature_pb2.Feature(\n",
        "                value_type=feature_pb2.Feature.ValueType.DOUBLE,\n",
        "                description=\"The average rating for the movie, range is [1.0-5.0]\",\n",
        "            ),\n",
        "            feature_id=\"average_rating\",\n",
        "        ),\n",
        "    ],\n",
        ").result()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xUVOzrAb1AFX"
      },
      "source": [
        "## Search created features\n",
        "\n",
        "While the [ListFeatures](https://cloud.google.com/vertex-ai/docs/featurestore/managing-featurestores#listing_featurestores) method allows you to easily view all features of a single\n",
        "entity type, the [SearchFeatures](https://cloud.google.com/vertex-ai/docs/matching-engine) method searches across all featurestores\n",
        "and entity types in a given location (such as `us-central1`). This can help you discover features that were created by someone else.\n",
        "\n",
        "You can query based on feature properties including feature ID, entity type ID,\n",
        "and feature description. You can also limit results by filtering on a specific\n",
        "featurestore, feature value type, and/or labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hs_7T_hs17ew",
        "outputId": "fc88c497-2503-4bd8-f6be-b864721b4689",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/average_rating\"\n",
              " description: \"The average rating for the movie, range is [1.0-5.0]\"\n",
              " create_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 36566000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 36566000\n",
              " },\n",
              " name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/genres\"\n",
              " description: \"The genres of the movie\"\n",
              " create_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 35305000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 35305000\n",
              " },\n",
              " name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/title\"\n",
              " description: \"The title of the movie\"\n",
              " create_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 32685000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 32685000\n",
              " },\n",
              " name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users/features/age\"\n",
              " description: \"User age\"\n",
              " create_time {\n",
              "   seconds: 1629803853\n",
              "   nanos: 449402000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803853\n",
              "   nanos: 449402000\n",
              " },\n",
              " name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users/features/gender\"\n",
              " description: \"User gender\"\n",
              " create_time {\n",
              "   seconds: 1629803853\n",
              "   nanos: 450760000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803853\n",
              "   nanos: 450760000\n",
              " },\n",
              " name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users/features/liked_genres\"\n",
              " description: \"An array of genres that this user liked\"\n",
              " create_time {\n",
              "   seconds: 1629803853\n",
              "   nanos: 451973000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803853\n",
              "   nanos: 451973000\n",
              " }]"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Search for all features across all featurestores.\n",
        "list(admin_client.search_features(location=BASE_RESOURCE_PATH))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jcxsiBUiIyvE"
      },
      "source": [
        "Now, narrow down the search to features that are of type `DOUBLE`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a9ovJSyEI4OZ",
        "outputId": "2b430858-5e9a-4521-ea5d-0a54cfedbde3",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/average_rating\"\n",
              " description: \"The average rating for the movie, range is [1.0-5.0]\"\n",
              " create_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 36566000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 36566000\n",
              " }]"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Search for all features with value type `DOUBLE`\n",
        "list(\n",
        "    admin_client.search_features(\n",
        "        featurestore_service_pb2.SearchFeaturesRequest(\n",
        "            location=BASE_RESOURCE_PATH, query=\"value_type=DOUBLE\"\n",
        "        )\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wtr9tvH6JAOY"
      },
      "source": [
        "Or, limit the search results to features with specific keywords in their ID and type."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3G1mNV1uJFBC",
        "outputId": "b9d7f9f9-3693-417d-b2be-11260db0b34c",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[name: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/movies/features/title\"\n",
              " description: \"The title of the movie\"\n",
              " create_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 32685000\n",
              " }\n",
              " update_time {\n",
              "   seconds: 1629803867\n",
              "   nanos: 32685000\n",
              " }]"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Filter on feature value type and keywords.\n",
        "list(\n",
        "    admin_client.search_features(\n",
        "        featurestore_service_pb2.SearchFeaturesRequest(\n",
        "            location=BASE_RESOURCE_PATH, query=\"feature_id:title AND value_type=STRING\"\n",
        "        )\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K3n5XdK8Xjmw"
      },
      "source": [
        "## Import Feature Values\n",
        "\n",
        "You need to import feature values before you can use them for online/offline serving. In this step, you will learn how to import feature values by calling the ImportFeatureValues API using the Python SDK.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BlqJ-QdTcs6W"
      },
      "source": [
        "### Source Data Format and Layout\n",
        "\n",
        "As mentioned above, BigQuery table/Avro/CSV are supported. No matter what format you are using, each imported entity *must* have an ID; also, each entity can *optionally* have a timestamp, specifying when the feature values are generated. This notebook uses Avro as an input, located at this public [bucket](https://pantheon.corp.google.com/storage/browser/cloud-samples-data/ai-platform-unified/datasets/featurestore;tab=objects?project=storage-samples&prefix=&forceOnObjectsSortingFiltering=false). The Avro schemas are as follows:\n",
        "\n",
        "**For the Users entity**:\n",
        "```\n",
        "schema = {\n",
        "  \"type\": \"record\",\n",
        "  \"name\": \"User\",\n",
        "  \"fields\": [\n",
        "      {\n",
        "       \"name\":\"user_id\",\n",
        "       \"type\":[\"null\",\"string\"]\n",
        "      },\n",
        "      {\n",
        "       \"name\":\"age\",\n",
        "       \"type\":[\"null\",\"long\"]\n",
        "      },\n",
        "      {\n",
        "       \"name\":\"gender\",\n",
        "       \"type\":[\"null\",\"string\"]\n",
        "      },\n",
        "      {\n",
        "       \"name\":\"liked_genres\",\n",
        "       \"type\":{\"type\":\"array\",\"items\":\"string\"}\n",
        "      },\n",
        "      {\n",
        "       \"name\":\"update_time\",\n",
        "       \"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]\n",
        "      },\n",
        "  ]\n",
        " }\n",
        "```\n",
        "\n",
        "**For the Movies entity**\n",
        "```\n",
        "schema = {\n",
        " \"type\": \"record\",\n",
        " \"name\": \"Movie\",\n",
        " \"fields\": [\n",
        "     {\n",
        "      \"name\":\"movie_id\",\n",
        "      \"type\":[\"null\",\"string\"]\n",
        "     },\n",
        "     {\n",
        "      \"name\":\"average_rating\",\n",
        "      \"type\":[\"null\",\"double\"]\n",
        "     },\n",
        "     {\n",
        "      \"name\":\"title\",\n",
        "      \"type\":[\"null\",\"string\"]\n",
        "     },\n",
        "     {\n",
        "      \"name\":\"genres\",\n",
        "      \"type\":[\"null\",\"string\"]\n",
        "     },\n",
        "     {\n",
        "      \"name\":\"update_time\",\n",
        "      \"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]\n",
        "     },\n",
        " ]\n",
        "}\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7DyDa6chbJx"
      },
      "source": [
        "### Import feature values for Users\n",
        "\n",
        "When importing, specify the following in your request:\n",
        "\n",
        "*   Data source format: BigQuery Table/Avro/CSV\n",
        "*   Data source URL\n",
        "*   Destination: featurestore/entity types/features to be imported\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "RUhm6-yzXjmx",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import_users_request = featurestore_service_pb2.ImportFeatureValuesRequest(\n",
        "    entity_type=admin_client.entity_type_path(\n",
        "        PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"\n",
        "    ),\n",
        "# TODO 1a\n",
        "# Make sure to replace your bucket name here.\n",
        "    avro_source=io_pb2.AvroSource(\n",
        "        # Source\n",
        "        gcs_source=io_pb2.GcsSource(\n",
        "            uris=[\n",
        "                \"gs://<Your-bucket-name>/users.avro\"\n",
        "            ]\n",
        "        )\n",
        "    ),\n",
        "    entity_id_field=\"user_id\",\n",
        "    feature_specs=[\n",
        "        # Features\n",
        "        featurestore_service_pb2.ImportFeatureValuesRequest.FeatureSpec(id=\"age\"),\n",
        "        featurestore_service_pb2.ImportFeatureValuesRequest.FeatureSpec(id=\"gender\"),\n",
        "        featurestore_service_pb2.ImportFeatureValuesRequest.FeatureSpec(\n",
        "            id=\"liked_genres\"\n",
        "        ),\n",
        "    ],\n",
        "    feature_time_field=\"update_time\",\n",
        "    worker_count=10,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "qwznuUiwjwJF",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Start to import, will take a couple of minutes\n",
        "ingestion_lro = admin_client.import_feature_values(import_users_request)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_sDl3ZcrF64T",
        "outputId": "005fbfe5-fce9-4309-8834-144799291bf5",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "imported_entity_count: 7\n",
              "imported_feature_value_count: 12"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Polls for the LRO status and prints when the LRO has completed\n",
        "ingestion_lro.result()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "laXdJPIqkLJO"
      },
      "source": [
        "### Import feature values for Movies\n",
        "\n",
        "Similarly, import feature values for 'movies' into the featurestore.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "-W1lCxgDl6iR",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import_movie_request = featurestore_service_pb2.ImportFeatureValuesRequest(\n",
        "    entity_type=admin_client.entity_type_path(\n",
        "        PROJECT_ID, REGION, FEATURESTORE_ID, \"movies\"\n",
        "    ),\n",
        "# TODO 1b\n",
        "# Make sure to replace your bucket name here.\n",
        "    avro_source=io_pb2.AvroSource(\n",
        "        gcs_source=io_pb2.GcsSource(\n",
        "            uris=[\n",
        "                \"gs://<Your-bucket-name>/movies.avro\"\n",
        "            ]\n",
        "        )\n",
        "    ),\n",
        "    entity_id_field=\"movie_id\",\n",
        "    feature_specs=[\n",
        "        featurestore_service_pb2.ImportFeatureValuesRequest.FeatureSpec(id=\"title\"),\n",
        "        featurestore_service_pb2.ImportFeatureValuesRequest.FeatureSpec(id=\"genres\"),\n",
        "        featurestore_service_pb2.ImportFeatureValuesRequest.FeatureSpec(\n",
        "            id=\"average_rating\"\n",
        "        ),\n",
        "    ],\n",
        "    feature_time_field=\"update_time\",\n",
        "    worker_count=10,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "Q-MATtpvm3HI",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Start to import, will take a couple of minutes\n",
        "ingestion_lro = admin_client.import_feature_values(import_movie_request)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VpaK3yRCnNnI",
        "outputId": "d239cab6-6eb9-4205-8e69-90b60d559793",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "imported_entity_count: 4\n",
              "imported_feature_value_count: 12"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Polls for the LRO status and prints when the LRO has completed\n",
        "ingestion_lro.result()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9TdxPYdDXjnA"
      },
      "source": [
        "## Online serving"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ezJIMyU-XjnB"
      },
      "source": [
        "The\n",
        "[Online Serving APIs](https://cloud.google.com/vertex-ai/docs/featurestore/serving-online)\n",
        "lets you serve feature values for small batches of entities. It's designed for latency-sensitive service, such as online model prediction. For example, for a movie service, you might want to quickly shows movies that the current user would most likely watch by using online predictions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "foNB0D2aw37c"
      },
      "source": [
        "### Read one entity per request\n",
        "\n",
        "The ReadFeatureValues API is used to read feature values of one entity; hence\n",
        "its custom HTTP verb is `readFeatureValues`. By default, the API will return the  latest value of each feature, meaning the feature values with the most recent  timestamp.\n",
        "\n",
        "To read feature values, specify the entity ID and features to read. The response\n",
        "contains a `header` and an `entity_view`. Each row of data in the `entity_view`\n",
        "contains one feature value, in the same order of features as listed in the response header."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3rfWqLrbXjnJ",
        "outputId": "460091eb-b08d-46d8-c518-6186460367db",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "header {\n",
              "  entity_type: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users\"\n",
              "  feature_descriptors {\n",
              "    id: \"age\"\n",
              "  }\n",
              "  feature_descriptors {\n",
              "    id: \"gender\"\n",
              "  }\n",
              "  feature_descriptors {\n",
              "    id: \"liked_genres\"\n",
              "  }\n",
              "}\n",
              "entity_view {\n",
              "  entity_id: \"alice\"\n",
              "  data {\n",
              "  }\n",
              "  data {\n",
              "  }\n",
              "  data {\n",
              "  }\n",
              "}"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Fetch the following 3 features.\n",
        "feature_selector = FeatureSelector(\n",
        "    id_matcher=IdMatcher(ids=[\"age\", \"gender\", \"liked_genres\"])\n",
        ")\n",
        "\n",
        "# TODO 2a\n",
        "data_client.read_feature_values(\n",
        "    featurestore_online_service_pb2.ReadFeatureValuesRequest(\n",
        "        # Fetch from the following feature store/entity type\n",
        "        entity_type=admin_client.entity_type_path(\n",
        "            PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"\n",
        "        ),\n",
        "        # Fetch the user features whose ID is \"alice\"\n",
        "        entity_id=\"alice\",\n",
        "        feature_selector=feature_selector,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oYk83Zt9xF8m"
      },
      "source": [
        "### Read multiple entities per request\n",
        "\n",
        "To read feature values from multiple entities, use the\n",
        "StreamingReadFeatureValues API, which is almost identical to the previous\n",
        "ReadFeatureValues API. Note that fetching only a small number of entities is recommended when using this API due to its latency-sensitive nature."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "BIJFcIIHULOd",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Read the same set of features as above, but for multiple entities.\n",
        "response_stream = data_client.streaming_read_feature_values(\n",
        "# TODO 2b\n",
        "    featurestore_online_service_pb2.StreamingReadFeatureValuesRequest(\n",
        "        entity_type=admin_client.entity_type_path(\n",
        "            PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"\n",
        "        ),\n",
        "        entity_ids=[\"alice\", \"bob\"],\n",
        "        feature_selector=feature_selector,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NFrVLiHyUj2l",
        "outputId": "1728ab80-fb1f-41d4-c4e1-dbaf7eeaf6b3",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "header {\n",
            "  entity_type: \"projects/1066331464660/locations/us-central1/featurestores/movie_prediction_20210824110930/entityTypes/users\"\n",
            "  feature_descriptors {\n",
            "    id: \"age\"\n",
            "  }\n",
            "  feature_descriptors {\n",
            "    id: \"gender\"\n",
            "  }\n",
            "  feature_descriptors {\n",
            "    id: \"liked_genres\"\n",
            "  }\n",
            "}\n",
            "\n",
            "entity_view {\n",
            "  entity_id: \"alice\"\n",
            "  data {\n",
            "  }\n",
            "  data {\n",
            "  }\n",
            "  data {\n",
            "  }\n",
            "}\n",
            "\n",
            "entity_view {\n",
            "  entity_id: \"bob\"\n",
            "  data {\n",
            "  }\n",
            "  data {\n",
            "  }\n",
            "  data {\n",
            "  }\n",
            "}\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Iterate and process response. Note the first one is always the header only.\n",
        "for response in response_stream:\n",
        "    print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sds42j8ZsCzS"
      },
      "source": [
        "Now that you have learned how to fetch imported feature values for online serving, the next step is learning how to use imported feature values for offline use cases."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpvhPAYxD-Ml"
      },
      "source": [
        "## Batch Serving\n",
        "\n",
        "Batch Serving is used to fetch a large batch of feature values for high-throughput, typically for training a model or batch prediction. In this section, you will learn how to prepare for training examples by calling the BatchReadFeatureValues API."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tPzAGvqJHh3B"
      },
      "source": [
        "### Use case\n",
        "\n",
        "**The task** is to prepare a training dataset to train a model, which predicts if a given user will watch a given movie. To achieve this, you need 2 sets of input:\n",
        "\n",
        "*   Features: you already imported into the featurestore.\n",
        "*   Labels: the groud-truth data recorded that user X has watched movie Y.\n",
        "\n",
        "\n",
        "To be more specific, the ground-truth observation is described in Table 1 and the desired training dataset is described in Table 2. Each row in Table 2 is a result of joining the imported feature values from Feature Store according to the entity IDs and timestamps in Table 1. In this example,  the `age`, `gender` and `liked_genres` features from `users` and\n",
        "the `genres` and `average_rating` features from `movies` are chosen to train the model. Note that only positive examples are shown in these 2 tables, i.e., you can imagine there is a label column whose values are all `True`.\n",
        "\n",
        "BatchReadFeatureValues API takes Table 1 as\n",
        "input, joins all required feature values from the featurestore, and returns Table 2 for training.\n",
        "\n",
        "<h4 align=\"center\">Table 1. Ground-truth Data</h4>\n",
        "\n",
        "users | movies | timestamp            \n",
        "----- | -------- | -------------------- \n",
        "alice  | Cinema Paradiso     | 2019-11-01T00:00:00Z \n",
        "bob  | The Shining     | 2019-11-15T18:09:43Z \n",
        "...   | ...      | ...     \n",
        "\n",
        "\n",
        "<h4 align=\"center\">Table 2. Expected Training Data Generated by Batch Read API (Positive Samples)</h4>\n",
        "\n",
        "timestamp            | entity_type_users | age | gender | liked_genres | entity_type_movies | genres | average_rating  \n",
        "-------------------- | ----------------- | --------------- | ---------------- | -------------------- | -------- | --------- | ----- \n",
        "2019-11-01T00:00:00Z | bob              | 35        | M                | [Action, Crime]                 | The Shining | Horror | 4.8 \n",
        "2019-11-01T00:00:00Z | alice             | 55        | F                | [Drama, Comedy]                 | Cinema Paradiso | Romance | 4.5 \n",
        "... | ... | ... | ... | ... | ... | ... | ... \n",
        "             "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hKwWGg2i4a4G"
      },
      "source": [
        "#### Why timestamp?\n",
        "\n",
        "Note that there is a `timestamp` column in Table 2. This indicates the time when the ground-truth was observed. This is to avoid data inconsistency.\n",
        "\n",
        "For example, the 1st row of Table 2 indicates that user `alice` watched movie `Cinema Paradiso` on `2019-11-01T00:00:00Z`. The featurestore keeps feature values for all timestamps but fetches feature values *only* at the given timestamp during batch serving. On 2019-11-01 alice might be 54 years old, but now alice might be 56; featurestore returns `age=54` as alice's age, instead of `age=56`, because that is the value of the feature at the observation time. Similarly, other features might be time-variant as well, such as liked_genres."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W8dLJ9nuDFgI"
      },
      "source": [
        "### Batch Read Feature Values\n",
        "\n",
        "Assemble the request which specify the following info:\n",
        "\n",
        "*   Where is the label data, i.e., Table 1.\n",
        "*   Which features are read, i.e., the column names in Table 2.\n",
        "\n",
        "The output is stored in a BigQuery table.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "7IyoXHY2ECnh",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "batch_serving_request = featurestore_service_pb2.BatchReadFeatureValuesRequest(\n",
        "    # featurestore info\n",
        "    featurestore=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
        "    # URL for the label data, i.e., Table 1.\n",
        "    csv_read_instances=io_pb2.CsvSource(\n",
        "        gcs_source=io_pb2.GcsSource(uris=[INPUT_CSV_FILE])\n",
        "    ),\n",
        "    destination=featurestore_service_pb2.FeatureValueDestination(\n",
        "        bigquery_destination=io_pb2.BigQueryDestination(\n",
        "            # Output to BigQuery table created earlier\n",
        "            output_uri=DESTINATION_TABLE_URI\n",
        "        )\n",
        "    ),\n",
        "    entity_type_specs=[\n",
        "        featurestore_service_pb2.BatchReadFeatureValuesRequest.EntityTypeSpec(\n",
        "            # Read the 'age', 'gender' and 'liked_genres' features from the 'users' entity\n",
        "            # TODO 3a\n",
        "            entity_type_id=\"users\",\n",
        "            feature_selector=FeatureSelector(\n",
        "                id_matcher=IdMatcher(\n",
        "                    ids=[\n",
        "                        # features, use \"*\" if you want to select all features within this entity type\n",
        "                        \"age\",\n",
        "                        \"gender\",\n",
        "                        \"liked_genres\",\n",
        "                    ]\n",
        "                )\n",
        "            ),\n",
        "        ),\n",
        "        featurestore_service_pb2.BatchReadFeatureValuesRequest.EntityTypeSpec(\n",
        "            # Read the 'average_rating' and 'genres' feature values of the 'movies' entity\n",
        "            # TODO 3b\n",
        "            entity_type_id=\"movies\",\n",
        "            feature_selector=FeatureSelector(\n",
        "                id_matcher=IdMatcher(ids=[\"average_rating\", \"genres\"])\n",
        "            ),\n",
        "        ),\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "bZO5sRCfEEWn",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Execute the batch read\n",
        "batch_serving_lro = admin_client.batch_read_feature_values(batch_serving_request)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ouMiJqh-EFlh",
        "outputId": "9408da49-092a-4ca3-cc89-a68e304b07fc",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# This long runing operation will poll until the batch read finishes.\n",
        "batch_serving_lro.result()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3RAdjahHQ93J"
      },
      "source": [
        "After the LRO finishes, you should be able to see the result from the [BigQuery console](https://console.cloud.google.com/bigquery), in the dataset created earlier."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "You can also keep the project but delete the featurestore:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a2WfqhbFzQBF",
        "outputId": "9296d758-8bd5-484b-fd4b-55a03ebbbf39",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Deleted dataset 'movie_predictions_20210824110930'.\n"
          ]
        }
      ],
      "source": [
        "admin_client.delete_featurestore(\n",
        "    request=featurestore_service_pb2.DeleteFeaturestoreRequest(\n",
        "        name=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
        "        force=True,\n",
        "    )\n",
        ").result()\n",
        "client.delete_dataset(\n",
        "    DESTINATION_DATA_SET, delete_contents=True, not_found_ok=True\n",
        ")  # Make an API request.\n",
        "\n",
        "print(\"Deleted dataset '{}'.\".format(DESTINATION_DATA_SET))"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "ze4-nDLfK4pw"
      ],
      "name": "6_gapic_feature_store.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
